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This paper shows the design and validation of a questionnaire aimed at college 
students to assess their satisfaction level with mathematics study materials. Starting 
from the theoretical framework presented, we proposed three dimensions: overall 
quality of mathematics study materials, didactic adequacy and motivation capacity. 
To that effect, we hereby explain the analysis and validation procedure of the 
psychometric properties of the assessment instrument. The study sample comprised 
1,666 university students. Sample was chosen using a random sampling technique. 
The exploratory factor analysis (EFA) and the confirmatory factor analysis (CFA) 
performed on two consecutive samples of freshmen studying Computer Science 
Engineering confirmed that the Questionnaire to assess student satisfaction with 
mathematics study materials measures satisfaction conditions in five scales: 
Didactic Adequacy, Relevance, Engagement, Interaction and Technological 
Quality. The results revealed the existence of significant psychometric features of 
the constructed questionnaire. 
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validation, quality of mathematics materials, student satisfaction 


INTRODUCTION 


The design of study materials according to students’ media literacy level is a highly 
topical and relevant reference in education research works (Ponce et al., 2019; West, 
2019; Zwart et al., 2017), and even more with the leading role acquired by digital 
education. This subject has become especially significant in very complex subjects such 
as mathematics, where students need to be motivated and satisfied with the education, 
they receive in order to achieve better academic results (Bicer & Capraro, 2019; 
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Putwain et al., 2018; Schenke et al., 2020). In this regard, in the course of our research 
we cannot stop wondering: how can we evaluate student satisfaction with the materials 
designed for the study of mathematics? The mediation performed by the materials 
designed and used by professors in the classroom is essential in order to funnel students’ 
learning; that is why we must know if students are satisfied with the materials designed 
by professors. 


In order to achieve this scientific goal, the literature review has provided us with a 
fragmented scientific outlook where differentiated traditions and approaches converge, 
together with different research lines. The first of them deals with the quality of 
mathematics study materials from an external and formal perspective (Kul et al., 2018; 
Shirai et al., 2018). The second one is related to the educational significance of the 
study materials used (Hadar, 2017; Lopez Hernandez et al., 2019). The last research line 
is focused on the role played by motivational variables to achieve a proper academic 
performance in students (Attard & Holmes, 2020; Fiorella et al., 2019). The 
aforementioned fragmentation becomes evident in the preparation of measurement tools 
as an expression of the research lines of a different nature in order to assess student 
satisfaction with study materials (Hansen & Gissel, 2017). 


With regard to the first research line proposed, there are works that, both from a 
theoretical and a methodological perspective, deal with the construct of the quality of 
mathematics materials from a multidimensional approach, integrating an evaluative 
approach in such analysis in order to get to know students’ perception in their capacity 
as main users. In this case, different variables have been associated with the quality of 
mathematics study materials. Research works (Onal, 2017; Sombra et al., 2019) suggest 
that graphic design, colors, image structure and contents have a direct impact on the 
quality of the materials. However, these research works do not contemplate that study 
materials provide individualized learning as a tool to contribute to learning satisfaction. 


Didactic adequacy in terms of significance, theoretical-practical connection and use 
difficulty level define another line of work; researchers like Gustiani et al. (2017) have 
concluded that study materials must be designed using a mathematical language 
attainable for students in order to contribute to mathematics learning. They have also 
underlined the fact that mathematical contents must be explained through 
understandable algorithms according to each student’s cognitive capacity (Hadar, 2017; 
Lazarides & Rubach, 2017); as well as the need of stressing the link between theory and 
practice, so that study materials may guide students through proper learning. 


The use of technologies to design study materials is another recurrent aspect in present 
research works (Lazem, 2019; West, 2019); establishing a clear relationship between 
satisfaction with mathematics study materials and their level of interactivity and 
feedback (Amaya et al., 2017; Rylands & Shearman, 2018). The latest research works 
advocate for the use of study materials linked to information and communication 
technologies (Abadi et al., 2017; Shirai et al., 2018; Zwart et al., 2017). These works 
provide important references on the quality expected for mathematics study materials in 
the strict sense. It is worth pointing out that, with regard to educational significance, 
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previous studies have not reflected the need of study materials to be highly illustrative 
with regard to mathematical definitions, theorems and postulates. 


The last research line on this subject includes the role played by motivational variables 
to achieve an adequate academic performance on students using materials that engage 
them in the learning process (Attard & Holmes, 2020; Durksen et al., 2017; Goldin, 
2018; Hidajat et al., 2020). Some of the main conclusions are which acceptable levels of 
engagement and interaction with study material are directly linked to the level of student 
satisfaction in their learning process (Attard & Holmes, 2020; Hodgson et al., 2017). 
This research does not demonstrate the existing relationship between syllabus and 
student engagement. 


Students’ active participation in their learning, as well as teaching quality self- 
perception, plays an essential role on mathematics education (Bassi, 2019). Student 
satisfaction levels have a direct impact on their academic performance in mathematics 
(Attard & Holmes, 2020; Y. Lee et al., 2019). As a result, the design of interactive 
educational materials adapted to students’ literacy level is a very relevant aspect from a 
scientific perspective in order to identify the desirable characteristics of any material in 
order to respond to the learning goals. On the basis of the above, figure | described the 
theoretical model proposed by us in order to implement the measurement instrument 
presented in this study. 


Overall quality segcreemceeeetttneneeetteeneenes + Didactic adequacy 
(Rul et al, 2018; Onal, 2017; Shirai (Gustiani et al, 2017; Hadar, 2017; 
et al, 2018; Sombra et al, 2019) Lazem, 2019; Lopez Hernandez et al, 
2019; West, 2019) 
Students’ 
7 satisfaction with Re 
bi mathematics ae 
study materials “eo 


“a POG 
Ld 


Motivational capacity 


(Attard & Holmes, 2020; Durksen et al, 
2017, Goldin, 2018; Hidajat et al, 2020, 
Norton, 2016) 


Figure 1 
Theoretical model about student satisfaction with mathematics study materials 


In short, we can conclude that the current scientific outlook is characterized by 
researches focused, on the one hand, on exploring students’ motivation with 
mathematics learning in general (Mercader et al., 2017; Straber, 2017), as well as on 
evidencing the impact of educational materials (Kul et al., 2018), without linking the 
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effect of both factors; few studies have worked from a multidimensional perspective to 
integrate the dimensions and variables analyzed in the different studies. Hence the need 
of designing assessment instruments that integrate the dimensions and variables that 
scientific literature is considering as relevant in the studies. 


Objectives and research hypothesis 


The main goal of the study is to get to know the psychometric properties of an 
instrument designed to measure student satisfaction with their mathematics study 
materials, specifically, their internal reliability and consistency (construct validation). In 
accordance with this objective, the hypotheses that guide this research have been 
described as follows: (H1) The instrument hereby presented has sufficient statistical 
consistency to be used to assess student satisfaction with their study materials and (H2) 
The factor analysis has confirmed the existence of the factors that explain student 
satisfaction with their mathematics study materials. 


Assessing student satisfaction with mathematics materials 


Starting from the theoretical proposal presented above, we proceeded to develop an 
instrument to assess student satisfaction with mathematics study materials. To that 
effect, we were inspired by two kinds of sources: empirical research on students’ 
motivation with mathematics learning and results of research works on the impact of 
educational materials on mathematics learning. 


The research work on students’ motivation with mathematics learning proves that study 
materials must be relevant, establishing for that purpose a link between the studies 
performed and the social context in which they have been developed (Bicer & Capraro, 
2019). It also stresses its utility to assess students (Matzakos & Kalogiannakis, 2017) 
and its capacity to provide wellbeing through study . 


Regarding the impact of educational materials in mathematics learning, we have paid 
special attention to the interaction and feedback (Cai et al., 2019) capacity of 
mathematics materials; in addition to technological quality, evidenced in the design 
(Hansen & Gissel, 2017) and use of audio-visual materials (Matzakos & Kalogiannakis, 
2017). 


The didactic adequacy of mathematics materials is especially relevant to foster students’ 
learning. These works agree that content contextualization, adjusted to the levels of 
difficulty for students, boost the level of satisfaction (Cai et al., 2019; Hansen & Gissel, 
2017). In addition, materials must guide students through mathematic learning 
including, among other aspects, a conceptual summary of the blocks of contents 
(Golding, 2018; Matzakos & Kalogiannakis, 2017). Table 1 synthetizes the 
aforementioned aspects. 


On the basis of the theoretical model studied, we proposed a preliminary questionnaire 
made up of 53 items grouped in three dimensions: overall quality (18 items), didactic 
adequacy (13 items) and motivational capacity (22 items). 
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Table 1 


Key concepts, related sources and sample items 


Key concepts 


Related sources 


Sample items 


Overall quality 


Audio-visual learning 
media 


(Matzakos & 
Kalogiannakis, 2017) 


Equipped with audio-visual resources 


Feedback 


(Abadi et al., 2017) 


Allow making questions to the author and 
receiving his/her answers. 


Interaction capacity 


(Amaya et al., 2017) 


Allow interaction with teachers and 
classmates. 


Didactic adequacy 


Theoretical abstracts 


(Matzakos & 
Kalogiannakis, 2017) 


Conceptual abstracts arranged by content 
blocks. 


Educational ; (Kul et al., 2018) Adjusted to the objectives and contents of 
contextualization the appropriate subjects. 

Adequate level of (Hansen & Gissel, Level of difficulty of theoretical contents 
difficulty 2017) according to the studies performed. 
Motivational capacity 

Utility for student (Matzakos & 

assessment Kalogiannakis, 2017) ie ed 
Relevance (Gustiani et al., 2017) Dini Contes ny 


Major/Degree/Studies. 


Capacity to provide 
wellbeing in the study 


(Garcia-Hernandez & 
Gonzalez-Ramirez, 


Foster a sense of wellbeing in the study of 
mathematics. 


2017) 


Purpose of the study 


The fundamental purpose of this work is to empirically validate the theoretical model 
presented here, starting with the data obtained using a questionnaire produced ‘ad hoc’ 
to Assess Student Satisfaction with Mathematics Study Materials. 


METHOD 
Designing a questionnaire 


Designing a questionnaire was supported by an exhaustive review of the literature on the 
construct “Student Satisfaction with Mathematics Study Materials”, and specifically on 
factors related to the educational contexts. The scale was structured into three 
dimensions that were developed into 53 items. These dimensions, presented in a 
previous work (Garcia-Hernandez & Gonzalez-Ramirez, 2018), are the following: 
didactic adequacy, motivational capacity and overall quality. 


Once designed, the instrument was subjected to a content validation process in different 
phases. In the first phase, the dimensions proposed and the questionnaire items were 
assessed by 7 experts in mathematics education. In each item, they assessed: (i) 
comprehension, referred to the level in which an item expresses its formulation in a 
specific and understandable manner, (ii) item feasibility, expressing the level in which 
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the item can be solved by students, and (iii) pertinence, referred to the level with which 
the item actually assesses student satisfaction with mathematics materials. 


We applied a qualitative content analysis to assess experts’ criteria. Therefore, we took 
into account the considerations that had coincidences in at least two judges. We 
modified the wording of items 4, 12, 21, 27, 42, 47, 56 and 65. We dropped items 25 
and 38 because they were already contained in item 5 and we also dropped items 7, 21, 
32 and 59 for being inappropriate to assess student satisfaction with mathematics study 
materials. At the end of this first phase, the questionnaire consisted of 47 items. 


Afterwards, we asked the other 10 experts from the sample to assess items in: essential 
items, useful (but not essential) items and unnecessary items. The index was calculated 
in accordance with the following formula: 


cvl =——= 


where ne is the number of experts who determine an item as essential and N is the total 
number of experts who took part in the process. Content Validity Index (CVI) is an 
index that fluctuates between -1 and 1, the higher the index the better the item 
assessment as essential. We think we need to use the CVI due to its relevance to assess 
the validity of the contents of the questionnaire and its convenience (as explained 
above). When the criterion has received a CVI=.80 by 10 experts or more, it is sufficient 
to determine that the item is essential (Almanasreh et al., 2017). From the total of 48 
items, 4 did not reach the CVI minimum value of 0.80. This way, four items were 
dropped by experts. Table 3 details the process of obtaining the CVI of the eliminated 
items. 


Table 2 

Process for obtaining the CVI of deleted items 

Expert Eliminated item 1 Eliminated item 2 Eliminated item 3 _ Eliminated item 4 
Expert | essential unnecessary useful useful 
Expert 2 essential essential essential unnecessary 
Expert 3 essential unnecessary essential unnecessary 
Expert 4 essential unnecessary essential unnecessary 
Expert 5 useful essential useful unnecessary 
Expert 6 essential essential useful essential 
Expert 7 essential essential useful useful 
Expert 8 unnecessary essential essential essential 
Expert 9 essential useful unnecessary essential 
Expert 10 unnecessary essential unnecessary unnecessary 
Experts in 

Agreement 

(essential) 7: 

(Ne) 

Item CVI __0.70 0.60 0.40 0.30 
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Finally, to build a more appropriate version of the questionnaire, we dropped four items, 
as their corrected homogeneity index was below .99. From these analyses, we obtained a 
questionnaire with 39 items grouped in three dimensions. For its application, we 
employed a Likert-type scale with values from 1 (strongly disagree) to 7 (strongly 
agree). The questionnaire was submitted online, with the consent of the University of 
Informatic Sciences, and the voluntary participation of freshmen studying Computer 
Science Engineering. 


Participants and characteristics of the sample 


In the construction and validation of instrument contents, there were three groups of 
participants at different times. The first sample was made up of a total of 17 teachers (11 
men and 6 women) through an intentional sampling selected from a population of 20 
professors of the Department of Mathematics of the Cuban University of Informatic 
Sciences (85.00%); specifically, in the subjects of linear algebra, continuous 
mathematics, discrete mathematics, probability and statistics. Convenience sampling 
was performed according to the following criteria: experts with a PhD academic degree 
and more than 12 years of experience teaching mathematics. 


In order to determine the reliability and validity of the questionnaire construct prepared, 
we recruited students of Computer Sciences Engineering (Universidad de Ciencias 
Informaticas, Cuba) for the survey. We incidentally selected freshmen, as they study 
four areas of mathematics: linear algebra, continuous mathematics, discrete mathematics 
I and discrete mathematics I. 


The exploratory validation stage counted on the participation of 728 students of the 746 
enrolled in the cohort of freshmen of the academic year 2017-2018. No sampling was 
carried out as the entire population participated in the study; however, eighteen students 
did not participate in the study because they were not present at the university for a long 
period of time (n=8) or because they declined the invitation voluntarily (n=10). The 
sample used for the confirmatory factor analysis was made up of 938 students out of 959 
enrolled in the cohort of freshmen of the academic year 2018-2019. Likewise, the entire 
population was taken into account, but 21 students did not participate in the survey 
because they were not present at the institution (n=6) or because they declined the 
invitation voluntarily (n=15). Table 2 shows a general outlook of both cohorts. 


Table 3 
Samples used for EFA (2017-2018) and CFA (2018-2019) 
Females in Males in Total Total Sample as % of total 
Cohort : ; 
sample sample sample population population 
2016-2017 293 435 728 746 97.59% 
2017-2018 377 561 938 959 97.81% 


Data analysis techniques 


First, we performed an exploratory factor analysis on cohort 2017-2018 in order to 
reduce its size. We employed the principal component procedure and the Varimax factor 
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rotation method. Previously, we used the Kaiser-Meyer-Olkin (KMO) test and Bartlett’s 
test for sphericity. For reliability calculation, we applied Cronbach’s coefficient alpha. 
The data analysis was performed using software IBM SPSS v22 and AMOS v18. 


Finally, we performed a confirmatory factor analysis from the responses of the students 
of cohort 2018-2019. To this effect, we determined model fit indicators: 


e Chi-square ratio on the degrees of freedom (y2/gl) to minimize the effects of the 
sample size. It is optimal if y2/gl takes a value between 2 and 3 with 
acceptability up to 5 (Gupta & Geetika, 2020). 


e Non-normalized fit index or Tucker-Lewis’s index (NNFI/TLI) and comparative 
fit index (CFI), compare the chi-square of the tested model with the theoretical 
model. This value, which is between 0 and 1, is considered acceptable when 
exceeds .90 (Shi et al., 2018). The CFI value, which may vary between 0 and 1, 
is the result of a comparison between the hypothetical model and a basic line 
model without correlations among any variables. 


e Root Mean Square Residual (SRMR) shows the mean size of residual 
correlations. Values between .5 and .8 are acceptable; indexes <.5 are regarded 
to be optimal (Pavlov et al., 2020). A nil value would mean that the model does 
not leave room for inexplicable standard residuals. 


e Root Mean Square Error Approximation (RMSEA) has been regarded as one of 
the strongest goodness of fit indexes; a value below .08 shall be regarded as 
acceptable, and below .05, as optimal (Gupta & Geetika, 2020). A perfect fit is 
represented by a nil value, which means that there would not be any difference 
between the variation observed and the model’s implicit variance. 


FINDINGS 


We shall expose the results obtained from the Exploratory Factor Analysis (EFA) and 
the Confirmatory Factor Analysis (CFA) performed. Both techniques have been widely 
used in studies on the development and validation of questionnaires as the one presented 
below. The EFA assumes that the variables observed are indicators of a certain number 
of common latent factors or variables. Assuming that we analyze a group of items 
selected to measure a single factor, each item analyzed has been carefully selected to 
reflect a characteristic of the factor that we intend to measure (Surastina & Dedi, 2018). 
The Confirmatory Factor Analysis (CFA) allows researchers to define how many factors 
are expected, which factors are related, and which items are related to each factor 
(Siewseng et al., 2021). 


Exploratory Factor Analysis 


To determine the internal validity of the questionnaire, we applied an EFA to the 39 
items of the questionnaire in order to analyze its internal structure. Before applying the 
EFA we employed the Kaiser-Meyer-Olkin’s test (KMO) and Bartlett’s sphericity test as 
statistical hypothesis. The result showed a KMO coefficient = .96, considered as very 
accepted and a Bartlett’s sphericity with a chi-square = 1489.98; and 5 degrees of 
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freedom; p <0.001. Results that evidence the feasibility of applying the EFA (Porto- 
Castro et al., 2018). 


Then we tried to obtain the factors applying the principal component method in order to 
obtain a group of components that explain the maximum total variance of original items. 
According to Kaiser’s normalization rule, that drops all components with eigenvalues 
under 1.0 (Gupta & Geetika, 2020), we obtained five factors that explain 68.64% of the 
variance. Next, we proceeded to remove five items whose commonality was below .50, 
as they did not explain the construct sufficiently and were not taken into account in the 
EFA’s final interpretation. 


After removing the aforementioned items, we proceeded to perform a new EFA (KMO 
.959 and Bartlett’s test for sphericity (chi-square = 1471.023; and 5 degrees of freedom; 
p <0.001). In this second analysis, we removed two items as their commonality was 
below .50. We performed a third analysis where definitively all commonalities were > 
.50. We maintained the initial factors with values > 1. This time, factors explain 73.37% 
of the variance, which demonstrates a 4.73% increase in relation to the initial analysis. 
The final results of the principal component factor extraction are shown in Appendix 1. 


The final version of the questionnaire consists of 32 items grouped in five factors; in 
Table 4 we can observe that none of the items has saturations < .45. Cronbach’s Alpha 
brought a result of .89, which proves a high reliability, thus conforming the first 
hypothesis (H1) of this research work. 


The first factor or component consists of 9 items that explain 39.48% of the variance. 
This dimension, that has been called didactic adequacy of mathematics study materials, 
is linked to the theoretical dimension of the same name and makes reference to the way 
in which the mathematic content is exposed, and the variability of the exercises 
developed, as well as the level of difficulty, among other aspects. This dimension has a 
Cronbach’s o of .83. 


The second and third factors are linked to the motivational capacity of mathematics 
study materials dimension. The second factor is made up of 10 items that explain 
13.73% of the variance and are related to the “Relevance” of the mathematics material 
for the student, understood as the utility of the learning provided, its application and the 
stimulation towards discovering a new knowledge. Cronbach’s a was .81. The third 
factor, made up of 4 items (8.00% of the variance), measures the level of “Engagement” 
that students can reach with the utilization of the study material to learn mathematics. 
This dimension takes into account the level of wellbeing reached by students with their 
materials and the study facilities provided. Cronbach’s a was.84. 
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Table 4 
General description of factor loadings of all the elements of the Questionnaire to assess 
student satisfaction with their mathematics study materials 


Items in the questionnaire’s order Fl F2  F3  F4~— FS 
Present mathematic contents in a pleasant way, showing their 49 

origin and evolution. : 

Conceptual summaries by content blocks. 48 


Present mathematic contents in a pleasant way, showing their 63 
origin and evolution. , 


Foster a feeling of wellbeing in the study of mathematics. .46 


It is possible to ask the author and receive his/her answers. .49 


Permit automatic self-evaluation. 51 


References to updated sources. 54 


Stimulate search and discovery. A9 


Connected with my interests through educational motivation 77 
activities, such as puzzles, riddles, etc. : 


Activities that promote game-based learning (gamification). .66 


Permit automatic self-evaluation. .60 


Adjusted to my speed of learning. 42 


Equipped with audio-visual resources. 47 


Contents linked to my degree/studies/major. 70 


Make visible the link between mathematic contents and real. 63 


Contribute to a better learning of the subject. 65 


Sufficient variety of exercises for my study. .49 


Accessible in mobile format. 64 


Attractive graphic design that encourages learning. .40 


Easy to understand and connected to my interests. 71 


Adapted to the subjects’ objectives and contents. 48 


The level of difficulty of theoretical contents is in accordance with 56 
my studies. ; 


Accurate progression of exercises regarding complexity. .49 


Adapted to my social and cultural reality. 5 


Permit interaction with teachers and classmates. 56 


Contents are updated every so often. 48 


Support the ideas or concepts developed in the text through 59 
illustrations or figures. : 


Help to forget how difficult mathematics are. 58 


Helpful in my academic exams. .66 


Describe algorithms step by step to understand mathematic .70 


Sufficient exemplification of mathematic definitions, theorems, 


and postulates. ae 


Adjusted to my personal interests. 49 
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The fourth and fifth factor are linked to the overall quality of mathematics study 
materials dimension. A total of 5 items integrates the fourth factor, explaining 6.45% of 
the variance. This dimension measures the “Interaction” capacity fostered by 
mathematics materials making reference, among other aspects, to the constant content 
updating, its capacity to clear up doubts and automated assessment. Cronbach’s a was 
.75. The fifth and last factor consists of 4 items that explain 5.71% of the variance. This 
dimension has been called “Technological quality” and includes graphic design, the 
utilization of illustrations and the possibility of providing audio-visual resources. 
Cronbach’s a was .69. 


Confirmatory Factor Analysis 


The model resulting from the EFA is made up of 5 factors, which correspond to the 
three dimensions of the theoretical model proposed for student satisfaction with their 
mathematics study materials. In line with this conceptual framework, we proceeded to 
correlate such five factors. The model’s initial fit was not acceptable, as indicated by fits 
y2/df = 3.21, CFI = .64, SRMR=.11 and RMSEA=.09. In order to improve the model 
fit, based on the highest standardized residuals, we removed three items. This gave rise 
to an improvement of the model, as reflected in the fit indexes: y2 / df = 1.39, CFI = .89, 
SRMR =.081, RMSEA = .052. The reliability of the five resulting factors expressed in 
Cronbach’s a was, respectively, .81 for DA, .83 for R, .77 for E, .78 for I and .66 for 
TQ (for a general description of factor loadings, check Table 5). 
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Table 5 
Factor loadings of remaining items of the student satisfaction with their mathematical 
materials Questionnaire after confirmatory factor analysis 


Item per hypothesized condition for student satisfaction with CFA factor loadings 


mathematics materials Fl F2. F3  F4_— F5 
Didactic adequacy of mathematics study materials 

Conceptual summaries by content blocks 48 
Present mathematic contents in a pleasant way, showing their 63 
origin and _ evolution : 
Sufficient variety of exercises for my study A9 
Adapted to the subjects’ objectives and contents A8 
The level of difficulty of theoretical contents is in accordance with 56 
my studies , 
Accurate progression of exercises regarding complexity A9 
Describe algorithms step by step to understand mathematic 70 
contents ; 
Sufficient exemplification of mathematic definitions, theorems 48 
and postulates : 


Motivational capacity of mathematics study materials 


Present mathematic contents in a pleasant way, showing their 


origin and evolution hid 


Stimulate search and discovery A9 


Connected with my interests through educational motivation 7 
activities, such as puzzles, riddles, etc. , 


Activities that promote game-based learning (gamification) .66 


Contents linked to my degree/studies/major 70 


Make visible the link between mathematic contents and real world 63 


Adapted to my social and cultural reality 5 


Helpful in my academic exams .66 


Foster a feeling of wellbeing in the study of mathematics 46 


Contribute to a better learning of the subject 65 


Easy to understand and connected to my interests 71 


Help to forget how difficult mathematics are 58 


Overall quality of mathematics study materials 


It is possible to ask the author and receive his/her answers .49 


Permit automatic self-evaluation .60 


Adjusted to my speed of learning 42 


Permit interaction with teachers and classmates 56 


Contents are updated every so often 48 


Equipped with audio-visual resources AT 


Accessible in mobile format 64 


Attractive graphic design that encourages learning 40 


Support the ideas or concepts developed in the text through 59 
illustrations or figures d 


Table 6 below shows correlation rates between model factors. As we can observe, 
correlations reach average values varying between .32 (between didactic adaptation and 
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relevance) and .76 (between technological quality and relevance). It is valid to explain 
that the lack of correlations between factors close to | dismisses the possibility of two 
factors representing the same dimension. This supports the tool validity, proving that it 
has different factors. 


Table 6 
Descriptives and correlations for the five scales of the student satisfaction with their 
mathematical materials Questionnaire after confirmatory factor analysis 


Scale Descriptives Pearson correlations 

Mean SD DA R E I TQ 
Didactic adequacy (DA) 4.67 1.02 1 7T6**  64** ADE SRE 
Relevance (R) 4.89 0.98 .76** 1 ADEE: OPER 3 2FR 
Engagement (E) 4.37 1.29 64** = 42** 1 48** 4Q** 
Interaction (1) 5.08 0.76 A2**. ST** . A8**. 1 ATE 
Technological quality (TQ) 4.44 1.12 8**  .32** 49** 47** 1 


Appendix 2 shows the results of the Confirmatory Factor Analysis (CFA) as the final 
phase of the validation of the designed questionnaire. 


DISCUSSION 


For many years, academic failure in mathematics has been a concern for the institutions 
and the international scientific community. Several investigations show that the overall 
quality and the didactic adequacy of study materials, together with student satisfaction, 
are factors that contribute to academic performance in mathematics. However, no 
studies were found that explain the relationship between the influence of these variables 
on mathematical learning, hence the importance of developing an instrument to assess 
student satisfaction with mathematical materials. 


At the end of the process of validation of the proposed questionnaire, the results have 
shown that this is a suitable and reliable instrument to obtain information on student 
satisfaction with mathematics materials. A conclusion drawn from the proposed 
methodological development, as well as by the validation process used, in accordance 
with the current trend of applying a sole, integrating process based on content and 
construction evidence (Gupta & Geetika, 2020). 


In the case of the validation of content, with the agreement of the judges and their 
contributions, it is possible to delimit and refine the specification matrix of the 
questionnaire, starting with 53 items. The elimination process carried out in this initial 
stage made it possible to obtain a 33-item questionnaire grouped in three dimensions. 


The validation of construct carried out, gives satisfactory results. The exploratory factor 
analysis showed the global existence of five dimensions or factors with a total of 29 
items, later ratified by confirmatory factor analysis showing an adequate adjustment to 
the proposed initial model. These factors coincided with the characteristics and 
theoretical foundations of the proposed dimensions: technological quality (Amaya et al., 
2017; Rylands & Shearman, 2018), didactic adaptation (Lazem, 2019; West, 2019), 
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engagement (Hansen & Gissel, 2017; Hodgson et al., 2017), relevance (Attard & 
Holmes, 2020; Bicer & Capraro, 2019) and interaction (Amaya et al., 2017; Lazem, 
2019). 


The aforementioned research has not considered satisfaction with Mathematics study 
materials from a multifactor perspective that integrates al the dimensions and variables 
addressed in the different studies. The results obtained in our research demonstrate the 
need of contributing to student satisfaction with their Mathematics materials from a 
global perspective integrating the three dimensions of the theoretical model proposed: 
educational suitability, motivational capacity and technological quality of the material; 
in order to foster interactivity in learning. 


This study entails a series of limitations that should be taken into account for future 
research. First, it is worth noting that the samples used for factor analysis represent a 
high percentage of the population, in both cases more than 97.5%; however, the students 
who took part in the study belong to a sole degree (Computer Science Engineering). For 
future studies, it would be advisable to incorporate students from different degrees to the 
study sample. 


Another limitation of this work is linked to the design of the questionnaire used. In 
future validations, it would also be necessary to include other analysis variables. As we 
have not found evidence of questionnaires that assess student satisfaction with their 
mathematics study materials, we must presume that different variables, which were not 
considered in this study, might be determinant in the assessment intended. This 
conceptual limitation should be taken into account to improve this questionnaire or to 
design future instruments. 


On a methodological level, we should also consider one aspect linked to the strength of 
the technological quality factor, derived from the EFA and the CFA. Its reliability is 
moderate with regard to the others (a = .69) and is made up of just four items; as a 
result, we need to reinforce this factor in terms of reliability and extension. It would be 
convenient to obtain an assessment of professor satisfaction with mathematics study 
materials in order to have other relevant criteria at our disposal for future research 
works. 


CONCLUSION 


This work analyzed the validity and reliability of a questionnaire to assess student 
satisfaction with their mathematics study materials. The psychometric scores were 
appropriate as we obtained values higher or equal to those recommended for tools 
designed to assess people satisfaction or perception. The realization of the Confirmatory 
Factor Analysis in a second cohort was the methodological procedure used to provide 
valid data considering the lack of research and the absence of questionnaires for 
comparative purposes. 


The attainment of the research hypotheses reinforces the tool potential as an expression 
of the theoretical model that integrates the study material’s dimensions of educational 
suitability, motivational capacity and technological quality, previously discussed in 
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scientific literature individually. This way we could provide professors and researchers 
with a useful and necessary instrument to get to know student satisfaction with their 
Mathematics study materials. 


SUGGESTIONS 


For future implementations, it would be advisable to complement it with a deeper study 
of a qualitative nature, as closed questions limit the students’ answers. Likewise, it 
would be interesting to get to know faculty’s satisfaction with their mathematics study 
materials. Finally, from a practical perspective, the results of applying this questionnaire 
shall provide relevant information for the design, selection and utilization of study 
materials in university institutions in order to improve students’ commitment and 
engagement towards mathematics learning. 
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APPENDIX 
Appendix 1 Results of the principal component factor extraction 
Initial eigenvalues Loading rotation square addition 
Component %, %, 
Total Vaiisies % Accumulated Total Warianes % Accumulated 
1 13.02 39.48 39.48 10.12 30.83 30.83 
2 5.27 13.73 53.21 7.00 15.50 46.33 
3 2.89 8.00 61.21 4.51 9.98 56.31 
4 1.64 6.45 67.66 2.96 9.31 65.62 
5 1.27 5.71 73.37 2.63 TIS 73.37 
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Appendix 2 Detailed results of the Confirmatory Factor Analysis (CFA) 
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